Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added dropped spans counter metric to catch number of dropped spans. #3386

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

rajat315315
Copy link
Contributor

Description

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

Fixes #3152

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

  • Test A

Does This PR Require a Contrib Repo Change?

Answer the following question based on these examples of changes that would require a Contrib Repo Change:

  • The OTel specification has changed which prompted this PR to update the method interfaces of opentelemetry-api/ or opentelemetry-sdk/

  • The method interfaces of test/util have changed

  • Scripts in scripts/ that were copied over to the Contrib repo have changed

  • Configuration files that were copied over to the Contrib repo have changed (when consistency between repositories is applicable) such as in

    • pyproject.toml
    • isort.cfg
    • .flake8
  • When a new .github/CODEOWNER is added

  • Major changes to project information, such as in:

    • README.md
    • CONTRIBUTING.md
  • Yes. - Link to PR:

  • No.

Checklist:

  • Followed the style guidelines of this project
  • Changelogs have been updated
  • Unit tests have been added
  • Documentation has been updated

@rajat315315 rajat315315 requested a review from a team July 19, 2023 12:06
self._dropped_counter = Meter.create_counter(
name="dropped_spans_counter", description="To count dropped Spans"
)
dropped_counter.add(1)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR needs testing to catch bugs like this one, the symbol dropped_counter is undefined.

@@ -228,7 +230,10 @@ def on_end(self, span: ReadableSpan) -> None:
if not self._spans_dropped:
logger.warning("Queue is full, likely spans will be dropped.")
self._spans_dropped = True

self._dropped_counter = Meter.create_counter(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Take a look at how this is being implemented for the Flask instrumentation.

@linux-foundation-easycla
Copy link

linux-foundation-easycla bot commented Aug 7, 2023

CLA Signed

The committers listed above are authorized under a signed CLA.

@rajat315315 rajat315315 force-pushed the dropped_counter branch 3 times, most recently from c865f16 to 290a993 Compare August 7, 2023 17:19
Signed-off-by: Rajat Jain <[email protected]>
@@ -228,7 +230,10 @@ def on_end(self, span: ReadableSpan) -> None:
if not self._spans_dropped:
logger.warning("Queue is full, likely spans will be dropped.")
self._spans_dropped = True

self._dropped_spans_counter = Meter.create_counter(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there's a bug here (not related to your change). self._spans_dropped should be reset to False after the queue has flushed some items. In this case, I would not rely on self._spans_dropped as the indicator of whether to create the counter, but instead if the counter exists already.

@@ -228,7 +230,10 @@ def on_end(self, span: ReadableSpan) -> None:
if not self._spans_dropped:
logger.warning("Queue is full, likely spans will be dropped.")
self._spans_dropped = True

self._dropped_spans_counter = Meter.create_counter(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tests are needed for this change.

@lzchen
Copy link
Contributor

lzchen commented Sep 6, 2023

@rajat315315
Are you still working on this?

@rajat315315
Copy link
Contributor Author

Extremely busy these days..
Cannot find time to complete this.

@ocelotl
Copy link
Contributor

ocelotl commented Mar 20, 2024

This PR needs further changes, marking it as draft.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Measure number of dropped spans in BatchSpanProcessor
3 participants